block: precalculate hashes if enabled and use them during compaction (downloading) #3031

GiedriusS · 2020-08-13T21:26:35Z

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

Added a change to objstore.DownloadDir so that it would be possible to ignore certain directories. Added functions for calculating the hashes of files and directories. Used them inside of shipper & other places. Wired up the new shipper changes and all of the other writers to command-line parameters so that the users can optionally enable them. There should be no performance impact if none of the flags (the default) have been enabled.

Verification

Unit tests. Manual testing. A new e2e test.

bwplotka

Nice, I love this generally and it gives further possibilities to check integrity. Some comments though (:

CHANGELOG.md

cmd/thanos/compact.go

pkg/block/block.go

pkg/block/metadata/hash.go

bwplotka · 2020-08-21T08:37:10Z

pkg/block/metadata/hash.go

+
+// NominalIgnoredFiles is a list of files that you want to typically ignore
+// when calculating the hashes of a TSDB block.
+var NominalIgnoredFiles = []string{DeletionMarkFilename, MetaFilename}


Well, the deletion mark could actually just be removed from here since it will never appear while a block is being produced. However, I think the meta.json file should be definitely excluded because there will be confusion because the hash will most likely change after adding it to m.Thanos.Hashes.

I mean, I would just rename it tbh or just use meta.json explicitly somewhere. Nominal does not mean anything to me 🤗

But overall it makes sense to skip hashing meta.json, it would hard... to achieve (:

pkg/compact/compact.go

pracucci

Thanks for working on this! The overall design LGTM. I agree with Bartek's comments plus I've added a couple of minor ones. Thanks!

pkg/objstore/objstore.go

pkg/block/block.go

pkg/block/metadata/hash.go

bwplotka

LGTM, just minor comments.

pkg/verifier/safe_delete.go

pkg/shipper/shipper.go

bwplotka · 2020-09-02T16:23:17Z

pkg/block/metadata/hash.go

+
+// NominalIgnoredFiles is a list of files that you want to typically ignore
+// when calculating the hashes of a TSDB block.
+var NominalIgnoredFiles = []string{DeletionMarkFilename, MetaFilename}


I mean, I would just rename it tbh or just use meta.json explicitly somewhere. Nominal does not mean anything to me 🤗

But overall it makes sense to skip hashing meta.json, it would hard... to achieve (:

onprem

Just some minor nits, everything else looks good to me!

onprem · 2020-09-05T13:40:19Z

cmd/thanos/receive.go

+	hashFunc := cmd.Flag("shipper.hash-func",
+		"Specify which hash function to use when calculating the hashes of produced files. If no function has been specified, it does not happen. This permits avoiding downloading some files twice albeit at some performance cost. Possible values are: \"\", \"SHA256\".").
+		Default("").Enum("", "SHA256")


As this flag is used in multiple places, maybe we should extract out it in a function in cmd/thanos/flags.go.

I agree, let's do it. However, how should we call it? Simply --hash-func? WDYT @prmsrswt, @bwplotka ?

I guess --hash-func would work.

pkg/block/metadata/meta.go

onprem · 2020-09-11T17:23:39Z

One more thing that came to my mind was, after using this feature, if someone decides to switch the hashing algorithm, al previous hashes would become unusable and we will trigger a redownload even though we have the correct files.

I guess we can also store the hashing algorithm used to create the hash alongside it so that we can calculate and match hashes using that algorithm irrespective of what algorithm is currently selected. This will also help us in identifying the hashing algorithm at places like Block Viewer and then showing some info to the user.

Quick Update: I saw that the hash struct has HashFunc as well, so we are already storing the function used to calculate the hash originally, but I don't think we are using that hash function during the pre-download check.

Apparently, I need to cross-check things before I put them here 🤦‍♂️ You are already doing what I suggested here.

stale · 2020-11-11T00:14:01Z

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

GiedriusS · 2020-11-19T07:01:50Z

Working on rebasing this.

GiedriusS · 2020-11-22T19:54:40Z

@bwplotka @prmsrswt @pracucci I have rebased this on master. PTAL! 💪 This version looks a bit cleaner to me because we now have that Files array in meta.json.

Added the possibility to ignore certain directories in objstore.{Download,DownloadDir}. Do not download files which have the same hash as in remote object storage. Wire up `--hash-func` so that writers could specify what hash function to use when uploading. There is no performance impact if no hash function has been explicitly specified. Clean up the removal of files logic in Thanos Compact to ensure we do not remove something that exists on disk already. Tested manually + new tests cover all of this more or less. Signed-off-by: Giedrius Statkevičius <[email protected]>

Signed-off-by: Giedrius Statkevičius <[email protected]>

This reverts commit 259c70b. Signed-off-by: Giedrius Statkevičius <[email protected]>

Signed-off-by: Giedrius Statkevičius <[email protected]>

pkg/block/metadata/hash.go

pkg/compact/compact.go

MalloZup

good work ! 👍 I just left some minor comment and took the occasion to review from curiosity pov to learn more the code .. 😁

Signed-off-by: Giedrius Statkevičius <[email protected]>

GiedriusS · 2021-02-23T20:21:05Z

@bwplotka @MalloZup I have fixed this PR according to your comments, added more extensive unit tests + added a e2e test which has caught some nasty bugs. I truly believe this is ready for prime time. PTAL! 🤗

bwplotka

Only minor logging nit, otherwise this looks good to me. Good tests and logic. 👍🏽 LGTM after this tiny log fix.

cmd/thanos/compact.go

bwplotka · 2021-02-24T18:28:29Z

cmd/thanos/downsample.go

 	if err := os.MkdirAll(dir, 0777); err != nil {
 		return errors.Wrap(err, "create dir")
 	}

 	defer func() {
+		// Leave the downsample directory for inspection if it is a halt error
+		// or if it is not then so that possibly we would not have to download everything again.
+		if rerr != nil {


What if it's disk full issue?

Then it's not a problem because we have a consistent order of work with #3843 in place +

thanos/pkg/compact/compact.go

Lines 312 to 314 in 6877bd1

sort.Slice(res, func(i, j int) bool {

return res[i].Key() < res[j].Key()

})

this exists already. This means that we are sure that after a retry, we are going to restart the work on the same blocks. If the blocks would disappear then accordingly we would get rid of them and then Thanos Compactor can continue its work. Does it make sense, @bwplotka ?

Makes sense 👍🏽

pkg/block/block.go

bwplotka · 2021-02-24T18:32:34Z

pkg/compact/compact.go

@@ -877,7 +884,10 @@ func NewBucketCompactor(
 // Compact runs compaction over bucket.
 func (c *BucketCompactor) Compact(ctx context.Context) (rerr error) {
 	defer func() {
-		if IsHaltError(rerr) {
+		// Do not remove the compactDir if an error has occurred


pkg/compact/compact.go

pkg/store/bucket_test.go

Signed-off-by: Giedrius Statkevičius <[email protected]>

bwplotka

LGTM!

Rebase and feel free to merge 🤗

Signed-off-by: Giedrius Statkevičius <[email protected]>

GiedriusS · 2021-03-01T12:54:55Z

🎉

I couldn't stop thinking about this code for some reason and I have figured that I had missed one case in thanos-io#3031. We need to also clean up the directories in compaction groups. A compaction could fail leaving some new directory with a random ULID on the disk. Before attempting to do another compaction loop, we need to remove it as well because the compaction process always produces a new, unique directory. Signed-off-by: Giedrius Statkevičius <[email protected]>

…(downloading) (thanos-io#3031) * block: precalculate hashes if enabled and use them during compaction Added the possibility to ignore certain directories in objstore.{Download,DownloadDir}. Do not download files which have the same hash as in remote object storage. Wire up `--hash-func` so that writers could specify what hash function to use when uploading. There is no performance impact if no hash function has been explicitly specified. Clean up the removal of files logic in Thanos Compact to ensure we do not remove something that exists on disk already. Tested manually + new tests cover all of this more or less. Signed-off-by: Giedrius Statkevičius <[email protected]> * block: expose GatherFileStats and use it Signed-off-by: Giedrius Statkevičius <[email protected]> * Revert "block: expose GatherFileStats and use it" This reverts commit 259c70b. Signed-off-by: Giedrius Statkevičius <[email protected]> * block: do not calc hash for dirs, add locks Signed-off-by: Giedrius Statkevičius <[email protected]> * docs/tools: update Signed-off-by: Giedrius Statkevičius <[email protected]> * shipper: pass s.hashFunc Signed-off-by: Giedrius Statkevičius <[email protected]> * Fix according to Bartek's comments Signed-off-by: Giedrius Statkevičius <[email protected]> * compact: clean up comment Signed-off-by: Giedrius Statkevičius <[email protected]> * block: close with log on error Signed-off-by: Giedrius Statkevičius <[email protected]> * *: remove unused FNs Signed-off-by: Giedrius Statkevičius <[email protected]> * compact: add e2e test for new hash functionality Signed-off-by: Giedrius Statkevičius <[email protected]> * Fix according to Bartek's comments Signed-off-by: Giedrius Statkevičius <[email protected]>

* compact: clean up directories properly I couldn't stop thinking about this code for some reason and I have figured that I had missed one case in #3031. We need to also clean up the directories in compaction groups. A compaction could fail leaving some new directory with a random ULID on the disk. Before attempting to do another compaction loop, we need to remove it as well because the compaction process always produces a new, unique directory. Signed-off-by: Giedrius Statkevičius <[email protected]> * Remove all non expected dirs. Signed-off-by: Bartlomiej Plotka <[email protected]> * Addressed comment. Signed-off-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]>

* tools: Fix partial and empty matchers in rewrite (#3891) * Fix partial and empty matchers match Signed-off-by: yeya24 <[email protected]> * add testcase for non-equal matchers Signed-off-by: yeya24 <[email protected]> * v0.19.0 patch: Added receive benchmark; Fixed Receiver excessive mem usage introduced in 0.17 (#3943) * Added receive benchmark, baseline. ``` goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them. BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./OK-12 22260 1550152 ns/op 1380340 B/op 6093 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./conflict_errors-12 6619 6430408 ns/op 4522487 B/op 26118 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them. BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./OK-12 2695 17208794 ns/op 15072963 B/op 60441 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./conflict_errors-12 474 72533286 ns/op 46396932 B/op 260141 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 270 137050518 ns/op 226595379 B/op 132 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 21 1616025443 ns/op 698724321 B/op 408 allocs/op PASS Process finished with exit code 0 ``` Signed-off-by: Bartlomiej Plotka <[email protected]> * Copy labels. ``` GOROOT=/home/bwplotka/.gvm/gos/go1.15 #gosetup GOPATH=/home/bwplotka/Repos/thanosgopath #gosetup /home/bwplotka/.gvm/gos/go1.15/bin/go test -c -o /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive github.com/thanos-io/thanos/pkg/receive #gosetup /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ -test.benchmem -test.benchtime=30s goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 25887 1537262 ns/op 1380023 B/op 6092 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 4237 7547968 ns/op 4522583 B/op 26118 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 2205 16513380 ns/op 15071092 B/op 60420 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 525 67278233 ns/op 46396645 B/op 260141 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 285 148049189 ns/op 226596168 B/op 132 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 20 1731361499 ns/op 698722550 B/op 401 allocs/op PASS Process finished with exit code 0 ``` Signed-off-by: Bartlomiej Plotka <[email protected]> * Addded bench., Signed-off-by: Bartlomiej Plotka <[email protected]> * Fix. Signed-off-by: Bartlomiej Plotka <[email protected]> * Improved API. Signed-off-by: Bartlomiej Plotka <[email protected]> * Changelog. Signed-off-by: Bartlomiej Plotka <[email protected]> * Addressed Lucas comments. Signed-off-by: Bartlomiej Plotka <[email protected]> # Conflicts: # CHANGELOG.md * compact: clean up directories thoroughly (#3869) * compact: clean up directories properly I couldn't stop thinking about this code for some reason and I have figured that I had missed one case in #3031. We need to also clean up the directories in compaction groups. A compaction could fail leaving some new directory with a random ULID on the disk. Before attempting to do another compaction loop, we need to remove it as well because the compaction process always produces a new, unique directory. Signed-off-by: Giedrius Statkevičius <[email protected]> * Remove all non expected dirs. Signed-off-by: Bartlomiej Plotka <[email protected]> * Addressed comment. Signed-off-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> * Added benchmark, Moved minio-deps to fork without race fix we don't need. (#3968) Fixes: #3917 Long term fix: #3967 Signed-off-by: Bartlomiej Plotka <[email protected]> # Conflicts: # go.sum * Changelog fix. Signed-off-by: Bartlomiej Plotka <[email protected]> * Build fix. Signed-off-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Ben Ye <[email protected]> Co-authored-by: Giedrius Statkevičius <[email protected]>

* Cut v0.19.0-rc.0 (#3860) Signed-off-by: Bartlomiej Plotka <[email protected]> * Fixed missed changelog comments. (#3865) Missed from: #3860 Signed-off-by: Bartlomiej Plotka <[email protected]> * rename Metadata API to MetricMetadata API (#3877) Signed-off-by: yeya24 <[email protected]> * Fix parseStep bug introduced in PR #3740 (#3887) (#3889) * Fix bug introduced in PR #3740 Signed-off-by: Hitanshu Mehta <[email protected]> * Minor change in test Signed-off-by: Hitanshu Mehta <[email protected]> Co-authored-by: Hitanshu Mehta <[email protected]> * Tools: rewrite delete can delete series only if it matches all matchers (#3886) (#3890) * rewrite delete should delete a series only if it matches all matchers in a deletion request Signed-off-by: yeya24 <[email protected]> * add test case Signed-off-by: yeya24 <[email protected]> Co-authored-by: Ben Ye <[email protected]> * cmd/thanos/receive.go: Receive client infers TLS (#3899) Currently, the thanos Receiver infers whether it should use TLS for the gRPC clients that forwards time series to other receivers based on whether the remote-write HTTP server uses TLS. This is not correct, as the HTTP server may use TLS without the gRPC server using TLS. This commit fixes the inference. Longer-term, this will be taken care of by the receive/router split. Signed-off-by: Lucas Servén Marín <[email protected]> * Cut v0.19.0-rc.1 (#3900) Signed-off-by: Bartlomiej Plotka <[email protected]> * tools: Fix partial and empty matchers in rewrite (#3891) * Fix partial and empty matchers match Signed-off-by: yeya24 <[email protected]> * add testcase for non-equal matchers Signed-off-by: yeya24 <[email protected]> * v0.19.0 patch: Added receive benchmark; Fixed Receiver excessive mem usage introduced in 0.17 (#3943) * Added receive benchmark, baseline. ``` goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them. BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./OK-12 22260 1550152 ns/op 1380340 B/op 6093 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./conflict_errors-12 6619 6430408 ns/op 4522487 B/op 26118 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them. BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./OK-12 2695 17208794 ns/op 15072963 B/op 60441 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./conflict_errors-12 474 72533286 ns/op 46396932 B/op 260141 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 270 137050518 ns/op 226595379 B/op 132 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 21 1616025443 ns/op 698724321 B/op 408 allocs/op PASS Process finished with exit code 0 ``` Signed-off-by: Bartlomiej Plotka <[email protected]> * Copy labels. ``` GOROOT=/home/bwplotka/.gvm/gos/go1.15 #gosetup GOPATH=/home/bwplotka/Repos/thanosgopath #gosetup /home/bwplotka/.gvm/gos/go1.15/bin/go test -c -o /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive github.com/thanos-io/thanos/pkg/receive #gosetup /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ -test.benchmem -test.benchtime=30s goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 25887 1537262 ns/op 1380023 B/op 6092 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 4237 7547968 ns/op 4522583 B/op 26118 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 2205 16513380 ns/op 15071092 B/op 60420 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 525 67278233 ns/op 46396645 B/op 260141 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 285 148049189 ns/op 226596168 B/op 132 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 20 1731361499 ns/op 698722550 B/op 401 allocs/op PASS Process finished with exit code 0 ``` Signed-off-by: Bartlomiej Plotka <[email protected]> * Addded bench., Signed-off-by: Bartlomiej Plotka <[email protected]> * Fix. Signed-off-by: Bartlomiej Plotka <[email protected]> * Improved API. Signed-off-by: Bartlomiej Plotka <[email protected]> * Changelog. Signed-off-by: Bartlomiej Plotka <[email protected]> * Addressed Lucas comments. Signed-off-by: Bartlomiej Plotka <[email protected]> * compact: clean up directories thoroughly (#3869) * compact: clean up directories properly I couldn't stop thinking about this code for some reason and I have figured that I had missed one case in #3031. We need to also clean up the directories in compaction groups. A compaction could fail leaving some new directory with a random ULID on the disk. Before attempting to do another compaction loop, we need to remove it as well because the compaction process always produces a new, unique directory. Signed-off-by: Giedrius Statkevičius <[email protected]> * Remove all non expected dirs. Signed-off-by: Bartlomiej Plotka <[email protected]> * Addressed comment. Signed-off-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> * Added benchmark, Moved minio-deps to fork without race fix we don't need. (#3968) Fixes: #3917 Long term fix: #3967 Signed-off-by: Bartlomiej Plotka <[email protected]> * Cut v0.19.0-rc.2 (#3969) Signed-off-by: Bartlomiej Plotka <[email protected]> * pkg/rules: fix deduplication of equal alerts with different labels (#3960) (#3999) Currently, if an alerting rule having the same name with different severity labels is being returned from different replicas then they are being treated as separate alerts. Given the following alerts a1,a2 with severities s1,s2 returned from replicas r1,2: a1[s1,r1] a1[s2,r1] a1[s1,r2] a1[s2,r2] Then, currently, the algorithm deduplicates to: a1[s1] a1[s2] a1[s1] a1[s2] Instead of the intendet result: a1[s1] a1[s2] This fixes it by removing replica labels before sorting labels for deduplication. Signed-off-by: Sergiusz Urbaniak <[email protected]> # Conflicts: # CHANGELOG.md Co-authored-by: Sergiusz Urbaniak <[email protected]> * Cut v0.19.0 (#3998) Signed-off-by: Bartlomiej Plotka <[email protected]> * Fixed VERSION file. Signed-off-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Ben Ye <[email protected]> Co-authored-by: Hitanshu Mehta <[email protected]> Co-authored-by: Lucas Servén Marín <[email protected]> Co-authored-by: Giedrius Statkevičius <[email protected]> Co-authored-by: Sergiusz Urbaniak <[email protected]>

@squat

* CHANGELOG.md: add v0.16.0 link (thanos-io#3697) This small commit fixes the changelog entry for release v0.16.0 so that the section title includes a link to the actual release. Signed-off-by: Lucas Servén Marín <[email protected]> * Delete deletion-mark.json at last when deleting a block (thanos-io#3661) * Delete deletion-mark.json at last when deleting a block Signed-off-by: Marco Pracucci <[email protected]> * Fixed linter Signed-off-by: Marco Pracucci <[email protected]> * fixed broken links (thanos-io#3645) Signed-off-by: Namanl2001 <[email protected]> * chore(tutorials): fix some broken links in 2-lts (thanos-io#3702) Signed-off-by: Bradley <[email protected]> * Fix race condition in BinaryReader.LookupSymbol() (thanos-io#3705) * Fix race condition in BinaryReader.LookupSymbol() Signed-off-by: Marco Pracucci <[email protected]> * Fixed BinaryReader receiver and added CHANGELOG entry Signed-off-by: Marco Pracucci <[email protected]> * ui: make old bucket viewer UI work with vanilla blocks (thanos-io#3700) * ui: make old UI work with vanilla blocks Make the old bucket viewer UI work with vanilla Prometheus blocks by checking whether the `Thanos` part exists before formatting the HTML. Signed-off-by: Giedrius Statkevičius <[email protected]> * CHANGELOG: update Signed-off-by: Giedrius Statkevičius <[email protected]> * ui/bucket: fix according to @squat's suggestions Signed-off-by: Giedrius Statkevičius <[email protected]> * ui: update bindata after latest changes Signed-off-by: Giedrius Statkevičius <[email protected]> * Query-frontend CMD: use detailed text description Signed-off-by: dmaiocchi <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> * Upgrade Cortex and remove SwiftConfig (thanos-io#3708) Signed-off-by: Marco Pracucci <[email protected]> * chore(deps): upgrading hugo (v0.80.0) fixes thanos-io#3653 (thanos-io#3714) Signed-off-by: Bradley <[email protected]> * store: Make more S3 http.Transport settings configurable. (thanos-io#3657) The http.Transport is not auto-tuning and one size does not seem to fit all cases. In order to respond to a query a store gateway might need to fetch a large (thousands) of postings, series and chunks. While the number of idle connections has been increased recently it can still be too low or expose bursty (opening, closing) behavior. Allow to tune most of the http.Transport parameters. I considered embedding the full Transport but there is already a Transport member. Signed-off-by: Holger Hans Peter Freyther <[email protected]> * added definiton of chunk and referenced it in the docs (thanos-io#3629) * Update design.md Update store.md Update design.md Update storage.md Update troubleshooting.md Signed-off-by: Biswajit Ghosh <[email protected]> * Update design.md Update store.md Update design.md Update troubleshooting.md Update storage.md Signed-off-by: Biswajit Ghosh <[email protected]> * Upgraded to newest bingo. (thanos-io#3718) Signed-off-by: Bartlomiej Plotka <[email protected]> * Update help messages with a consistent use of capitals and periods. (thanos-io#3727) * Update help messages with a consistent use of capitals and periods. Signed-off-by: Matt Whitney <[email protected]> * Update documentation to reflect message changes. Signed-off-by: Matt Whitney <[email protected]> * Run `make docs` to correct the documentation rather than manual edits. Signed-off-by: Matt Whitney <[email protected]> * docs: update examples/dashboards/dashboards.md (thanos-io#3733) Signed-off-by: Mert Acikportali <[email protected]> * pkg/errutil: correct the multierror file suffix (thanos-io#3736) This commit changes the file suffix from .go.go to simply .go. Signed-off-by: Lucas Servén Marín <[email protected]> * Fixed website and added new step for release process (tmp). (thanos-io#3738) Signed-off-by: Bartlomiej Plotka <[email protected]> * api: Added Proposal for Discovery Endpoint idea (InfoAPI) (thanos-io#3703) * docs/proposals/210701_endpoint_discovery.md: Add initial proposal Signed-off-by: Lili Cosic <[email protected]> * Update docs/proposals/210701_endpoint_discovery.md Co-authored-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> * Introduce "block-meta-fetch-concurrency" flag for compact and store (thanos-io#3752) * Introduce "block-meta-fetch-concurrency" flag for compact and store Furthermore consolidate the fetcher concurrency by moving the fetcherConcurrency const to the fetcher code of the block package. Signed-off-by: Johannes Frey <[email protected]> * Document flags Signed-off-by: Johannes Frey <[email protected]> * Add `--query-range.request-downsampled` flag to Query Frontend (thanos-io#2641) (thanos-io#3723) * Add `--query-range.request-downsampled` flag to Query Frontend (thanos-io#2641) Signed-off-by: Vladimir Kononov <[email protected]> * Apply suggestions from code review Co-authored-by: Bartlomiej Plotka <[email protected]> Signed-off-by: Vladimir Kononov <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> * Pinned to newer busybox on quay. (thanos-io#3762) Signed-off-by: Bartlomiej Plotka <[email protected]> * Fix panic on concurrent index-header lazy reader usage and unload (thanos-io#3760) * Fix panic on concurrent index-header lazy reader usage and unload Signed-off-by: Marco Pracucci <[email protected]> * Addressed review comments Signed-off-by: Marco Pracucci <[email protected]> * Addressed review comments Signed-off-by: Marco Pracucci <[email protected]> * Fixed Planner.Plan() description (thanos-io#3767) Signed-off-by: Marco Pracucci <[email protected]> * Fixed panic on concurrent index-header lazy load / unload (thanos-io#3759) Signed-off-by: Marco Pracucci <[email protected]> * Pad compaction planner size check (thanos-io#3773) Pad the index size check in the compaction planner by 15% to avoid situations where the sum of index bytes bloats to be larger than the sum of the original index sizes. * thanos-io#3724 * thanos-io#3750 Signed-off-by: Ben Kochie <[email protected]> * pkg/server/http: Allow passing http.ServeMux with a server option (thanos-io#3769) Signed-off-by: Matthias Loibl <[email protected]> * Update Katacoda Official Link (thanos-io#3768) Signed-off-by: soniasingla <[email protected]> * Nomitate kakkoyun for the v0.20.0 release (thanos-io#3778) Signed-off-by: Kemal Akkoyun <[email protected]> * Fix typo in query component documentation for store api link (thanos-io#3675) * Fix wrong store api link Signed-off-by: Junyoung, Sung <[email protected]> * Fix store api link to absolute Signed-off-by: Junyoung, Sung <[email protected]> * e2e: Resolved Flaky tests (thanos-io#3777) * added retry function Signed-off-by: Abhishek357 <[email protected]> * fixed CI test Signed-off-by: Abhishek357 <[email protected]> * restructured code Signed-off-by: Abhishek357 <[email protected]> * Add an exempt label to ignore issue that has a PR Signed-off-by: Kemal Akkoyun <[email protected]> * in memory cache for caching bucket (thanos-io#3579) * in memeory cache for caching bucket Signed-off-by: Sudharshann D <[email protected]> * addressing comments Signed-off-by: Sudharshann D <[email protected]> * Introduce "allow overlapping blocks" flag for Thanos receiver (thanos-io#3792) * add tsdb.allow-overlapping-blocks flag to receiver Signed-off-by: Max Chandler <[email protected]> * make format Signed-off-by: Max Chandler <[email protected]> * cleanup whitespace Signed-off-by: Max Chandler <[email protected]> * cleanup whitespace Signed-off-by: Max Chandler <[email protected]> * make docs Signed-off-by: Max Chandler <[email protected]> * fix changelog conflict Signed-off-by: Max Chandler <[email protected]> Co-authored-by: Matt Lawrence <[email protected]> * Reduce memory allocations in bucketBlock.readChunkRange() (thanos-io#3796) Signed-off-by: Marco Pracucci <[email protected]> * update prometheus version Signed-off-by: Mauro Stettler <[email protected]> * update cortex Signed-off-by: Mauro Stettler <[email protected]> * add new property to LabelValuesRequest Signed-off-by: Mauro Stettler <[email protected]> * set label matchers in LabelValueRequest Signed-off-by: Mauro Stettler <[email protected]> * make docs Signed-off-by: Mauro Stettler <[email protected]> * Allow to customise S3 SSE on a per-request basis (thanos-io#3783) * Allow to customise S3 SSE on a per-request basis Signed-off-by: Marco Pracucci <[email protected]> * Addressed review comments Signed-off-by: Marco Pracucci <[email protected]> * Refactoring: allow to pass BytesPool to store.NewBucketStore() (thanos-io#3801) Signed-off-by: Marco Pracucci <[email protected]> * Allow to customise the partitioner used by the BucketStore (thanos-io#3802) Signed-off-by: Marco Pracucci <[email protected]> * Fix button display when there is no panels (thanos-io#3694) Signed-off-by: Namanl2001 <[email protected]> * fix tests Signed-off-by: Mauro Stettler <[email protected]> * Allow downstream projects to customise the Partitioner (thanos-io#3808) Signed-off-by: Marco Pracucci <[email protected]> * Makefile: Fix to command to find React source files (thanos-io#3805) Signed-off-by: Hitanshu Mehta <[email protected]> * Merge release 0.18 (thanos-io#3809) * CHANGELOG.md: release v0.18.0 Signed-off-by: Lucas Servén Marín <[email protected]> * VERSION,tutorials: bump versions Signed-off-by: Lucas Servén Marín <[email protected]> * Revert "VERSION,tutorials: bump versions" This reverts commit 5a27c70. The "thanos:" prefix for the container images was accidentally removed. Signed-off-by: Lucas Servén Marín <[email protected]> * VERSION,tutorials: bump version for all images Signed-off-by: Lucas Servén Marín <[email protected]> * CHANGELOG: bump release date Signed-off-by: Lucas Servén Marín <[email protected]> * pkg/rules/proxy: fix hotlooping when receiving client errors Currently, if we receive an error from the underlying client stream, we continue with trying to receive additional data. This causes a hotloop as we will receive the same error again. This fixes it by returning in the error case and adds a unit test for the proxy logic. Fixes thanos-io#3717 Signed-off-by: Sergiusz Urbaniak <[email protected]> * CHANGELOG.md: fix changelog to incorporate new fix (thanos-io#3737) This commit fixes the order of the changelog to properly reflect a recent change. Signed-off-by: Lucas Servén Marín <[email protected]> * Fixed website and added new step for release process (tmp). (thanos-io#3738) Signed-off-by: Bartlomiej Plotka <[email protected]> * CHANGELOG.md: update v0.18.0 release date Signed-off-by: Lucas Servén Marín <[email protected]> Co-authored-by: Sergiusz Urbaniak <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> * reloader: try to fix test flakiness (thanos-io#3798) This is an attempt of fixing the `TestReloader_DirectoriesApply` test flakiness. I call this an attempt because I cannot reproduce it locally. However, I have noticed that during runs where this fails the logs look like this: ``` --- FAIL: TestReloader_DirectoriesApply (3.04s) reloader_test.go:256: Performing step number 0 reloader_test.go:256: Performing step number 1 reloader_test.go:256: Performing step number 2 reloader_test.go:256: Performing step number 3 reloader_test.go:256: Performing step number 4 reloader_test.go:256: Performing step number 6 reloader_test.go:343: reloader_test.go:343: exp: 5 got: 6 ``` It immediately jumps to another value. This gave me a hint and I think that this is happening because potentially `i` can be written to/read from by multiple goroutines. On very resource constrained systems like the CircleCI runners, it could just happen that the `if` doesn't do what it is supposed to. Try to fix this problem by protecting the whole HTTP handler with a mutex. Since this is only a test and not a performance critical path, I think this is a reasonable change to do. Add extra check for reload failures. Signed-off-by: Giedrius Statkevičius <[email protected]> * addressing PR comments Signed-off-by: Mauro Stettler <[email protected]> * Block Viewer: Move overlapping blocks to separate rows (thanos-io#3729) * Block Viewer: overlapping blocks issue resolved Signed-off-by: Namanl2001 <[email protected]> * small changes and make assets Signed-off-by: Namanl2001 <[email protected]> * Add flag to set default step Signed-off-by: Hitanshu Mehta <[email protected]> * Add support to use default step in classic ui Signed-off-by: Hitanshu Mehta <[email protected]> * Add support to use default step in new ui Signed-off-by: Hitanshu Mehta <[email protected]> * Lint fixes Signed-off-by: Hitanshu Mehta <[email protected]> * change ParseStep function to be consistent with ui Signed-off-by: Hitanshu Mehta <[email protected]> * Improve description of query.default-step flag Signed-off-by: Hitanshu Mehta <[email protected]> * Minor fixes Signed-off-by: Hitanshu Mehta <[email protected]> * Use default value when flag is undefined Signed-off-by: Hitanshu Mehta <[email protected]> * minor fixes Signed-off-by: Hitanshu Mehta <[email protected]> * Minor fixes Signed-off-by: Hitanshu Mehta <[email protected]> * Update changelog Signed-off-by: Hitanshu Mehta <[email protected]> * Fix typo Signed-off-by: Hitanshu Mehta <[email protected]> * Receive: Improve handling of empty time series from clients (thanos-io#3815) * exit early if a request has no timeseries data Signed-off-by: Matt Lawrence <[email protected]> Co-authored-by: Max Chandler <[email protected]> * update changelog Signed-off-by: Matt Lawrence <[email protected]> * update changelog Signed-off-by: Matt Lawrence <[email protected]> * Move constant to right side of comparison Co-authored-by: Lucas Servén Marín <[email protected]> Signed-off-by: Matt Lawrence <[email protected]> Co-authored-by: Max Chandler <[email protected]> Co-authored-by: Lucas Servén Marín <[email protected]> * Reduced allocated memory by chunks reader in the store gateway at query time (thanos-io#3814) * Reduced allocated memory by chunks reader in the store gateway at query time Signed-off-by: Marco Pracucci <[email protected]> * Fixed linter issues Signed-off-by: Marco Pracucci <[email protected]> * Fixed linter (hopefully) Signed-off-by: Marco Pracucci <[email protected]> * Renamed function Signed-off-by: Marco Pracucci <[email protected]> * Updated comment Signed-off-by: Marco Pracucci <[email protected]> * Updated code comment Signed-off-by: Marco Pracucci <[email protected]> * promclient: fix error's message (thanos-io#3824) Use the provided `method` in the error messages. I got scared reading the logs that the requests are still being sent using GET instead of POST which I had specified. Signed-off-by: Giedrius Statkevičius <[email protected]> * pkg/ui/react-app: update dependencies (thanos-io#3818) Fix lodash security issue. Signed-off-by: Simon Pasquier <[email protected]> * Add objstore.List() recursive support (thanos-io#3823) * Add objstore.List() recursive support Signed-off-by: Marco Pracucci <[email protected]> * Fixed CachingBucket Signed-off-by: Marco Pracucci <[email protected]> * Fix Cortex compilation issue Signed-off-by: Marco Pracucci <[email protected]> * Tidy go.mod/sum Signed-off-by: Marco Pracucci <[email protected]> * Fix linter Signed-off-by: Marco Pracucci <[email protected]> * Fixed TestMetricBucket_Close Signed-off-by: Marco Pracucci <[email protected]> * Upgraded Cortex Signed-off-by: Marco Pracucci <[email protected]> * Inlined struct initialisation Signed-off-by: Marco Pracucci <[email protected]> * Upgrade Cortex again Signed-off-by: Marco Pracucci <[email protected]> * Upgrade Cortex (thanos-io#3828) Signed-off-by: Marco Pracucci <[email protected]> * Truncated S3 "get object" response is reported as error (thanos-io#3795) * Add test to ensure S3 truncated response is an error Signed-off-by: Marco Pracucci <[email protected]> * Added missing copyright Signed-off-by: Marco Pracucci <[email protected]> * Upgraded Minio Signed-off-by: Marco Pracucci <[email protected]> * Upgraded Minio again Signed-off-by: Marco Pracucci <[email protected]> * Implement federated metric metadata API (thanos-io#3686) * support federated metadata API Signed-off-by: Ben Ye <[email protected]> * update comments Signed-off-by: yeya24 <[email protected]> * use parseInt Signed-off-by: yeya24 <[email protected]> * address Prem's comments Signed-off-by: yeya24 <[email protected]> * update proto comment Signed-off-by: yeya24 <[email protected]> * add changelog Signed-off-by: yeya24 <[email protected]> * Fixed TestBucketStore_ManyParts_e2e (thanos-io#3841) https://app.circleci.com/pipelines/github/thanos-io/thanos/5120/workflows/efd2a21d-13b7-4035-99e3-cb1af8023694/jobs/13809 This fail is only visible for anyone from Thanos Team proposing PR, due to extra tests against bucket providers. Signed-off-by: Bartlomiej Plotka <[email protected]> * Reinforcing errutil and fixed ugly bug which caused empty []error to te treated as error, (thanos-io#3836) Previous multi-error implementation could cause very ugly bug of returnig empty multi-error that should be treated as success not error by API, but if .Err() is not invoked it will be used as non nil error. Once we merge this, we can do cleaner solution that slighly change nesting behaviour: thanos-io#3833 There were 9 places where we had this bug in handler due to MultiError lib allowing to do so. Signed-off-by: Bartlomiej Plotka <[email protected]> * Export metadata fetcher metrics (thanos-io#3660) Signed-off-by: Marco Pracucci <[email protected]> * store: Cleaned up API for test/benchmark purposes. (thanos-io#3650) Signed-off-by: Bartlomiej Plotka <[email protected]> * Allow Cortex to fully reuse FetcherMetrics (thanos-io#3842) Signed-off-by: Marco Pracucci <[email protected]> * downsample: ensure consistent order (thanos-io#3843) Ensure that we have a consistent order of blocks that we are going to downsample. `range` over maps doesn't enforce any particular order on purpose. This is needed for https://github.com/thanos-io/thanos/pull/3031/files. ATM in that PR before downsampling we delete all directories which do not match blocks ULIDs in the remote object storage. Ideally, we should only keep around the files of a block which we are about to downsample. It is impossible to do that properly ATM if during another iteration we'd start from a different block. Thus, let's have a consistent order. Signed-off-by: Giedrius Statkevičius <[email protected]> * [Fix] Replace unsupported %v macro in logs (thanos-io#3847) Signed-off-by: Stéphane Brillant <[email protected]> * Update ui npm dependencies to latest versions (thanos-io#3813) * Update ui npm deps to latest Signed-off-by: Saswata Mukherjee <[email protected]> * Fix linting errors Signed-off-by: Saswata Mukherjee <[email protected]> * Change camelcase lint rule Signed-off-by: Saswata Mukherjee <[email protected]> * Fix CI configs and scripts to work with main branch (thanos-io#3848) Signed-off-by: Prem Saraswat <[email protected]> * block: precalculate hashes if enabled and use them during compaction (downloading) (thanos-io#3031) * block: precalculate hashes if enabled and use them during compaction Added the possibility to ignore certain directories in objstore.{Download,DownloadDir}. Do not download files which have the same hash as in remote object storage. Wire up `--hash-func` so that writers could specify what hash function to use when uploading. There is no performance impact if no hash function has been explicitly specified. Clean up the removal of files logic in Thanos Compact to ensure we do not remove something that exists on disk already. Tested manually + new tests cover all of this more or less. Signed-off-by: Giedrius Statkevičius <[email protected]> * block: expose GatherFileStats and use it Signed-off-by: Giedrius Statkevičius <[email protected]> * Revert "block: expose GatherFileStats and use it" This reverts commit 259c70b. Signed-off-by: Giedrius Statkevičius <[email protected]> * block: do not calc hash for dirs, add locks Signed-off-by: Giedrius Statkevičius <[email protected]> * docs/tools: update Signed-off-by: Giedrius Statkevičius <[email protected]> * shipper: pass s.hashFunc Signed-off-by: Giedrius Statkevičius <[email protected]> * Fix according to Bartek's comments Signed-off-by: Giedrius Statkevičius <[email protected]> * compact: clean up comment Signed-off-by: Giedrius Statkevičius <[email protected]> * block: close with log on error Signed-off-by: Giedrius Statkevičius <[email protected]> * *: remove unused FNs Signed-off-by: Giedrius Statkevičius <[email protected]> * compact: add e2e test for new hash functionality Signed-off-by: Giedrius Statkevičius <[email protected]> * Fix according to Bartek's comments Signed-off-by: Giedrius Statkevičius <[email protected]> * mixin: Upgrade jsonnet tooling (thanos-io#3855) * Upgrade jsonnet tooling Signed-off-by: Kemal Akkoyun <[email protected]> * Fix bingo version issue Signed-off-by: Kemal Akkoyun <[email protected]> * Fix unintended removal issue Signed-off-by: Kemal Akkoyun <[email protected]> * Update reference of master to main in docs (thanos-io#3849) * Update reference of master to main in docs Signed-off-by: Prem Saraswat <[email protected]> * Update image tag in tutorials/kubernetes-helm Signed-off-by: Prem Saraswat <[email protected]> * Cut v0.19.0-rc.0 (thanos-io#3860) Signed-off-by: Bartlomiej Plotka <[email protected]> * Fixed missed changelog comments. (thanos-io#3865) Missed from: thanos-io#3860 Signed-off-by: Bartlomiej Plotka <[email protected]> * rename Metadata API to MetricMetadata API (thanos-io#3877) Signed-off-by: yeya24 <[email protected]> * Fix parseStep bug introduced in PR thanos-io#3740 (thanos-io#3887) (thanos-io#3889) * Fix bug introduced in PR thanos-io#3740 Signed-off-by: Hitanshu Mehta <[email protected]> * Minor change in test Signed-off-by: Hitanshu Mehta <[email protected]> Co-authored-by: Hitanshu Mehta <[email protected]> * Tools: rewrite delete can delete series only if it matches all matchers (thanos-io#3886) (thanos-io#3890) * rewrite delete should delete a series only if it matches all matchers in a deletion request Signed-off-by: yeya24 <[email protected]> * add test case Signed-off-by: yeya24 <[email protected]> Co-authored-by: Ben Ye <[email protected]> * cmd/thanos/receive.go: Receive client infers TLS (thanos-io#3899) Currently, the thanos Receiver infers whether it should use TLS for the gRPC clients that forwards time series to other receivers based on whether the remote-write HTTP server uses TLS. This is not correct, as the HTTP server may use TLS without the gRPC server using TLS. This commit fixes the inference. Longer-term, this will be taken care of by the receive/router split. Signed-off-by: Lucas Servén Marín <[email protected]> * Cut v0.19.0-rc.1 (thanos-io#3900) Signed-off-by: Bartlomiej Plotka <[email protected]> * tools: Fix partial and empty matchers in rewrite (thanos-io#3891) * Fix partial and empty matchers match Signed-off-by: yeya24 <[email protected]> * add testcase for non-equal matchers Signed-off-by: yeya24 <[email protected]> * v0.19.0 patch: Added receive benchmark; Fixed Receiver excessive mem usage introduced in 0.17 (thanos-io#3943) * Added receive benchmark, baseline. ``` goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them. BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./OK-12 22260 1550152 ns/op 1380340 B/op 6093 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them./conflict_errors-12 6619 6430408 ns/op 4522487 B/op 26118 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them. BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./OK-12 2695 17208794 ns/op 15072963 B/op 60441 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them./conflict_errors-12 474 72533286 ns/op 46396932 B/op 260141 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 270 137050518 ns/op 226595379 B/op 132 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 21 1616025443 ns/op 698724321 B/op 408 allocs/op PASS Process finished with exit code 0 ``` Signed-off-by: Bartlomiej Plotka <[email protected]> * Copy labels. ``` GOROOT=/home/bwplotka/.gvm/gos/go1.15 #gosetup GOPATH=/home/bwplotka/Repos/thanosgopath #gosetup /home/bwplotka/.gvm/gos/go1.15/bin/go test -c -o /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive github.com/thanos-io/thanos/pkg/receive #gosetup /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ -test.benchmem -test.benchtime=30s goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 25887 1537262 ns/op 1380023 B/op 6092 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 4237 7547968 ns/op 4522583 B/op 26118 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 2205 16513380 ns/op 15071092 B/op 60420 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 525 67278233 ns/op 46396645 B/op 260141 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 285 148049189 ns/op 226596168 B/op 132 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 20 1731361499 ns/op 698722550 B/op 401 allocs/op PASS Process finished with exit code 0 ``` Signed-off-by: Bartlomiej Plotka <[email protected]> * Addded bench., Signed-off-by: Bartlomiej Plotka <[email protected]> * Fix. Signed-off-by: Bartlomiej Plotka <[email protected]> * Improved API. Signed-off-by: Bartlomiej Plotka <[email protected]> * Changelog. Signed-off-by: Bartlomiej Plotka <[email protected]> * Addressed Lucas comments. Signed-off-by: Bartlomiej Plotka <[email protected]> * compact: clean up directories thoroughly (thanos-io#3869) * compact: clean up directories properly I couldn't stop thinking about this code for some reason and I have figured that I had missed one case in thanos-io#3031. We need to also clean up the directories in compaction groups. A compaction could fail leaving some new directory with a random ULID on the disk. Before attempting to do another compaction loop, we need to remove it as well because the compaction process always produces a new, unique directory. Signed-off-by: Giedrius Statkevičius <[email protected]> * Remove all non expected dirs. Signed-off-by: Bartlomiej Plotka <[email protected]> * Addressed comment. Signed-off-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> * Added benchmark, Moved minio-deps to fork without race fix we don't need. (thanos-io#3968) Fixes: thanos-io#3917 Long term fix: thanos-io#3967 Signed-off-by: Bartlomiej Plotka <[email protected]> * Cut v0.19.0-rc.2 (thanos-io#3969) Signed-off-by: Bartlomiej Plotka <[email protected]> * pkg/rules: fix deduplication of equal alerts with different labels (thanos-io#3960) (thanos-io#3999) Currently, if an alerting rule having the same name with different severity labels is being returned from different replicas then they are being treated as separate alerts. Given the following alerts a1,a2 with severities s1,s2 returned from replicas r1,2: a1[s1,r1] a1[s2,r1] a1[s1,r2] a1[s2,r2] Then, currently, the algorithm deduplicates to: a1[s1] a1[s2] a1[s1] a1[s2] Instead of the intendet result: a1[s1] a1[s2] This fixes it by removing replica labels before sorting labels for deduplication. Signed-off-by: Sergiusz Urbaniak <[email protected]> # Conflicts: # CHANGELOG.md Co-authored-by: Sergiusz Urbaniak <[email protected]> * Cut v0.19.0 (thanos-io#3998) Signed-off-by: Bartlomiej Plotka <[email protected]> * Fixed VERSION file. Signed-off-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Lucas Servén Marín <[email protected]> Co-authored-by: Marco Pracucci <[email protected]> Co-authored-by: Naman Lakhwani <[email protected]> Co-authored-by: Bradley <[email protected]> Co-authored-by: Giedrius Statkevičius <[email protected]> Co-authored-by: dmaiocchi <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> Co-authored-by: Ben Ye <[email protected]> Co-authored-by: Holger Freyther <[email protected]> Co-authored-by: Biswajit Ghosh <[email protected]> Co-authored-by: Matt W <[email protected]> Co-authored-by: Mert Açıkportalı <[email protected]> Co-authored-by: Lili Cosic <[email protected]> Co-authored-by: Johannes Frey <[email protected]> Co-authored-by: Vladimir Kononov <[email protected]> Co-authored-by: Ben Kochie <[email protected]> Co-authored-by: Matthias Loibl <[email protected]> Co-authored-by: Sonia Singla <[email protected]> Co-authored-by: Kemal Akkoyun <[email protected]> Co-authored-by: Junyoung, Sung <[email protected]> Co-authored-by: Abhishek Singh Chauhan <[email protected]> Co-authored-by: Sudhar287 <[email protected]> Co-authored-by: Max Chandler <[email protected]> Co-authored-by: Matt Lawrence <[email protected]> Co-authored-by: Mauro Stettler <[email protected]> Co-authored-by: Hitanshu Mehta <[email protected]> Co-authored-by: Sergiusz Urbaniak <[email protected]> Co-authored-by: Hitanshu Mehta <[email protected]> Co-authored-by: Matt Lawrence <[email protected]> Co-authored-by: Max Chandler <[email protected]> Co-authored-by: Simon Pasquier <[email protected]> Co-authored-by: Stephane Brillant <[email protected]> Co-authored-by: Saswata Mukherjee <[email protected]> Co-authored-by: Prem Saraswat <[email protected]>

GiedriusS force-pushed the add_hash_method branch from 62d16cc to 86abed4 Compare August 18, 2020 20:21

GiedriusS changed the title ~~[WIP] block: precalculate hashes if enabled and use them during compaction (downloading)~~ block: precalculate hashes if enabled and use them during compaction (downloading) Aug 18, 2020

GiedriusS force-pushed the add_hash_method branch from 7171510 to efb42d8 Compare August 18, 2020 21:14

GiedriusS marked this pull request as ready for review August 18, 2020 21:15

GiedriusS mentioned this pull request Aug 19, 2020

Improving {Halt,Retry}Error semantics #2784

Closed

bwplotka reviewed Aug 21, 2020

View reviewed changes

pracucci reviewed Aug 25, 2020

View reviewed changes

pkg/objstore/objstore.go Outdated Show resolved Hide resolved

pkg/block/block.go Outdated Show resolved Hide resolved

pkg/block/metadata/hash.go Outdated Show resolved Hide resolved

bwplotka approved these changes Sep 2, 2020

View reviewed changes

GiedriusS force-pushed the add_hash_method branch from 5bf85b7 to 95bf117 Compare September 5, 2020 10:33

onprem approved these changes Sep 5, 2020

View reviewed changes

stale bot added the stale label Nov 11, 2020

stale bot closed this Nov 19, 2020

GiedriusS reopened this Nov 19, 2020

stale bot removed the stale label Nov 19, 2020

GiedriusS force-pushed the add_hash_method branch 4 times, most recently from 85c07fa to b5448f2 Compare November 22, 2020 19:40

GiedriusS force-pushed the add_hash_method branch from b5448f2 to 8bae707 Compare November 22, 2020 21:10

GiedriusS added 3 commits December 2, 2020 18:39

Merge remote-tracking branch 'origin/master' into add_hash_method

4ca451f

Signed-off-by: Giedrius Statkevičius <[email protected]>

block: expose GatherFileStats and use it

259c70b

Signed-off-by: Giedrius Statkevičius <[email protected]>

Merge remote-tracking branch 'origin/master' into add_hash_method

a938561

Signed-off-by: Giedrius Statkevičius <[email protected]>

onprem self-requested a review December 18, 2020 18:34

GiedriusS added 2 commits December 22, 2020 21:55

Revert "block: expose GatherFileStats and use it"

e8023b8

This reverts commit 259c70b. Signed-off-by: Giedrius Statkevičius <[email protected]>

block: do not calc hash for dirs, add locks

6e8ebc8

Signed-off-by: Giedrius Statkevičius <[email protected]>

MalloZup reviewed Dec 29, 2020

View reviewed changes

pkg/block/metadata/hash.go Outdated Show resolved Hide resolved

MalloZup reviewed Dec 29, 2020

View reviewed changes

pkg/compact/compact.go Outdated Show resolved Hide resolved

MalloZup reviewed Dec 29, 2020

View reviewed changes

GiedriusS added 6 commits February 23, 2021 11:46

Merge remote-tracking branch 'origin/master' into add_hash_method

0bd177a

Signed-off-by: Giedrius Statkevičius <[email protected]>

Fix according to Bartek's comments

43dd78b

Signed-off-by: Giedrius Statkevičius <[email protected]>

compact: clean up comment

e8fac48

Signed-off-by: Giedrius Statkevičius <[email protected]>

block: close with log on error

fab4562

Signed-off-by: Giedrius Statkevičius <[email protected]>

*: remove unused FNs

883550f

Signed-off-by: Giedrius Statkevičius <[email protected]>

compact: add e2e test for new hash functionality

cad16e4

Signed-off-by: Giedrius Statkevičius <[email protected]>

GiedriusS requested a review from bwplotka February 23, 2021 20:21

bwplotka approved these changes Feb 24, 2021

View reviewed changes

Fix according to Bartek's comments

fd00457

Signed-off-by: Giedrius Statkevičius <[email protected]>

Base automatically changed from master to main February 26, 2021 16:30

bwplotka approved these changes Mar 1, 2021

View reviewed changes

Merge remote-tracking branch 'origin/main' into add_hash_method

2175598

Signed-off-by: Giedrius Statkevičius <[email protected]>

GiedriusS merged commit 1bbad3b into thanos-io:main Mar 1, 2021

GiedriusS mentioned this pull request Mar 3, 2021

compact: clean up directories thoroughly #3869

Merged

2 tasks

bwplotka mentioned this pull request Mar 10, 2021

Improve Objstore Client Level Failures and Retries #3907

Closed

GiedriusS mentioned this pull request Mar 23, 2021

objstore.DownloadDir() doesn't correctly cleanup on failure #3941

Closed

yeya24 mentioned this pull request Nov 19, 2021

Tools bucket verify must validate chunks #4881

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

block: precalculate hashes if enabled and use them during compaction (downloading) #3031

block: precalculate hashes if enabled and use them during compaction (downloading) #3031

GiedriusS commented Aug 13, 2020 •

edited

Loading

bwplotka left a comment

bwplotka Aug 21, 2020

GiedriusS Aug 31, 2020

bwplotka Sep 2, 2020

pracucci left a comment

bwplotka left a comment

bwplotka Sep 2, 2020

onprem left a comment

onprem Sep 5, 2020

GiedriusS Sep 6, 2020

onprem Sep 11, 2020

onprem commented Sep 11, 2020 •

edited

Loading

stale bot commented Nov 11, 2020

GiedriusS commented Nov 19, 2020

GiedriusS commented Nov 22, 2020

MalloZup left a comment

GiedriusS commented Feb 23, 2021

bwplotka left a comment

bwplotka Feb 24, 2021

GiedriusS Feb 28, 2021

bwplotka Mar 1, 2021

bwplotka Feb 24, 2021

bwplotka left a comment

GiedriusS commented Mar 1, 2021

	sort.Slice(res, func(i, j int) bool {
	return res[i].Key() < res[j].Key()
	})

block: precalculate hashes if enabled and use them during compaction (downloading) #3031

block: precalculate hashes if enabled and use them during compaction (downloading) #3031

Conversation

GiedriusS commented Aug 13, 2020 • edited Loading

Changes

Verification

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pracucci left a comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

onprem left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

onprem commented Sep 11, 2020 • edited Loading

stale bot commented Nov 11, 2020

GiedriusS commented Nov 19, 2020

GiedriusS commented Nov 22, 2020

MalloZup left a comment

Choose a reason for hiding this comment

GiedriusS commented Feb 23, 2021

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

GiedriusS commented Mar 1, 2021

GiedriusS commented Aug 13, 2020 •

edited

Loading

onprem commented Sep 11, 2020 •

edited

Loading